Input-dependent Regularization of Conditional Density Models

We emphasize the need for input-dependent regularization in the context of conditional density models (also: discriminative models) like Gaussian process predictors. This can be achieved by a simple modification of the standard Bayesian data generation model underlying these techniques. Specifically, we allow the latent target function to be apriori dependent on the distribution of the input points. While the standard generation model results in robust predictors, data with missing labels is ignored, which can be wasteful if relevant prior knowledge is available. We show that discriminative models like Fisher kernel discriminants and CoTraining classifiers can be regarded as (approximate) Bayesian inference techniques under the modified generation model, and that the template Co-Training algorithm is related to a variant of the well-known ExpectationMaximization (EM) technique. We propose a template EM algorithm for the modified generation model which can be regarded as generalization of Co-Training.